Brief intro to R

# Load libraries
library(tidyverse)
library(robis) # retrieves open source species occurrence data
library(biomod2) # species distribution modeling package
library(terra)
library(raster)
# R basics ----
# Assignment operators
x <- 2 # preferred
x = 2 # also works

# Math operations
x * 2
## [1] 4
x^2 == x**2
## [1] TRUE
x/10
## [1] 0.2
1e+12
## [1] 1e+12
log(pi)
## [1] 1.14473
log10(pi)
## [1] 0.4971499
# Useful tip -- pipes
# create some data
x <- c(1, 2, 3, 4, 5, 6) # Create a vector
x <- 1:6 # alternate notation
# Compute mean of x
mean(x)
## [1] 3.5
# Alternative with piping
x |> mean()
## [1] 3.5
# Data frame example with made up data
fake.zoop.data <- data.frame("Genus" = c("Calanus", "Temora", "Centropages"),
                           "Species" = c("finmarchicus", "longicornis", "typicus"),
                           "Abundance" = c(250, 450, 500),
                           "NumAdults" = c(175, 300, 450),
                           "MeanDryWeight" = c(17, 10, 8),
                           "StErrDryWeight" = c(3.4, 1.6, 0.5))

# View data frame
View(fake.zoop.data)

# Accessing data frame columns
fake.zoop.data$Genus
## [1] "Calanus"     "Temora"      "Centropages"
# Accessing multiple columns
fake.zoop.data[c("Genus", "Species")]
##         Genus      Species
## 1     Calanus finmarchicus
## 2      Temora  longicornis
## 3 Centropages      typicus
# Add a column (base R)
fake.zoop.data$NumJuvenile <- c(75, 150, 50)

# Modifying a column
fake.zoop.data$NumJuvenile <- fake.zoop.data$Abundance - fake.zoop.data$NumAdults

We can do similar data manipulation using more intuitive methods from the dplyr package. The dplyr package is part of the larger tidyverse, which is also worth familiarizing yourself with if you program in R a lot. There are many helpful cheat sheets available. Here are a few examples below:

# Add a column to original data frame
fake.zoop.data <- fake.zoop.data |>
  dplyr::mutate(SampMethod = c("Vertical", "Vertical", "Vertical"))

fake.zoop.data
##         Genus      Species Abundance NumAdults MeanDryWeight StErrDryWeight
## 1     Calanus finmarchicus       250       175            17            3.4
## 2      Temora  longicornis       450       300            10            1.6
## 3 Centropages      typicus       500       450             8            0.5
##   NumJuvenile SampMethod
## 1          75   Vertical
## 2         150   Vertical
## 3          50   Vertical
# Filter the data frame
one.genus <- fake.zoop.data |>
  dplyr::filter(Genus == "Calanus" )

one.genus
##     Genus      Species Abundance NumAdults MeanDryWeight StErrDryWeight
## 1 Calanus finmarchicus       250       175            17            3.4
##   NumJuvenile SampMethod
## 1          75   Vertical
# Can combine these functions calls in a pipe
one.species <- fake.zoop.data |>
  dplyr::mutate(SampMethod = c("Vertical", "Vertical", "Vertical")) |>
  dplyr::filter(Genus == "Calanus" ) |>
  dplyr::select(Genus, Abundance, SampMethod)

one.species
##     Genus Abundance SampMethod
## 1 Calanus       250   Vertical

Next, we’ll plot the fake data in base R.

# Plot the data
plot(x = fake.zoop.data$Abundance, y = fake.zoop.data$MeanDryWeight)

The base R plot function works fine, but there are also plotting packages with a lot more flexibility and features.

One example is the ggplot2 package, also from the tidyverse:

# A better way to plot the data
ggplot(data = fake.zoop.data, mapping = aes(x = Abundance, y = MeanDryWeight)) +
  geom_point()

# Alternative notation
ggplot() +
  geom_point(data = fake.zoop.data, mapping = aes(x = Abundance, y = MeanDryWeight))

# Add error bars
ggplot(data = fake.zoop.data, mapping = aes(x = Abundance, y = MeanDryWeight)) +
  geom_point() + 
  geom_errorbar(mapping = aes(ymin = MeanDryWeight-StErrDryWeight, ymax = MeanDryWeight+StErrDryWeight))

# Improve plot aesthetics
ggplot(data = fake.zoop.data, mapping = aes(x = Abundance, y = MeanDryWeight)) +
  geom_point() + 
  geom_errorbar(mapping = aes(ymin = MeanDryWeight-StErrDryWeight, ymax = MeanDryWeight+StErrDryWeight)) +
  labs(x = bquote("Abundance (individuals"~m^{-3}*")"), y = "Mean Dry Weight (mg)") +
  theme_bw() # many theme options

Species distribution modeling in R

Species distribution models (SDMs) are an important tool used often in ecology to model habitat suitability for a given species. There are many ways to produce SDMs in R. Below is a simple example using the package biomod2, which has the capability to simultaneously produce multiple types of models and build ensembles. This package can be a good introduction to SDMs if you are new to coding and/or modeling. Biomod2 can produce models using up to 11 different algorithms. It runs the models behind the scenes using the respective R packages, which makes it a great package for introducing SDMs and quickly comparing different models using relatively little code.

The biomod2 github page is a really helpful resource: [Ensemble Platform for Species Distribution Modeling][https://biomodhub.github.io/biomod2/]

Fetch species data

The first step is fetching the species data. For this example, I will be using right whale occurence data from the Ocean Biodiversity Information System (OBIS), which is a global open access resource for marine biodiversity data and information.

# Fetching species data
RightWhale <- robis::occurrence("Eubalaena glacialis", # Right whale
                                startdate = as.Date("2005-01-01"),
                                enddate = as.Date("2015-12-31"))
## Retrieved 1549 records of approximately 1549 (100%)

Pulling records from OBIS for right whales between 2005 and 2015 resulted in 1,549 data points. Next, we will visualize our species data using ggplot2.

# Load world map data 
worldmap <- ggplot2::map_data("world")

# Plot occurrences on world map
ggplot2::ggplot(data = RightWhale, mapping = aes(x = decimalLongitude, y = decimalLatitude)) +
  # Add occurrence data
  geom_point() +
  # Add map data
  geom_polygon(data = worldmap, aes(long, lat, group = group), fill = NA, colour = "gray43") +
  # Sets map bounds to geographic range of the species data
  coord_quickmap(xlim = c(round(min(RightWhale$decimalLongitude)), 
                          round(max(RightWhale$decimalLongitude))), 
                 ylim = c(round(min(RightWhale$decimalLatitude)), 
                          round(max(RightWhale$decimalLatitude))),
                 expand = TRUE) +
  # Clean up theme
  theme_bw() +
  theme(panel.grid.major = element_blank(), panel.grid.minor = element_blank())

We can see that the data are largely concentrated along the east coast of the United States, with one point in the Azores. Since the environmental data we will use for modeling are regional and only for the summer, we need to crop the data to that bounding box and select only data from July through September.

# Define bounding box for data -- zoom in on Gulf of Maine
BB <- c(-75, -60, 35, 46)

# Crop data to bounding box
RightWhale <- RightWhale |>
  # Shorten names
  dplyr::rename(lon = decimalLongitude,
                lat = decimalLatitude) |>
  dplyr::mutate(year = as.numeric(year),
                month = as.numeric(month)) |>
  # Filter July, Aug, Sep
  dplyr::filter(month %in% 7:9) |>
  # Filter to bounding box
  dplyr::filter(lon >= BB[1] & lon <= BB[2] &
                  lat >= BB[3] & lat <= BB[4]) |>
  # Select relevant columns
  dplyr::select(lon, lat, year)
  
# Plot again
ggplot2::ggplot(data = RightWhale, mapping = aes(x = lon, y = lat)) +
  # Add occurrence data
  geom_point() +
  # Add map data
  geom_polygon(data = worldmap, aes(long, lat, group = group), fill = NA, colour = "gray43") +
  coord_quickmap(xlim = c(BB[1], BB[2]), 
                 ylim = c(BB[3], BB[4]),
                 expand = TRUE) +
  # Clean up theme
  theme_bw() +
  theme(panel.grid.major = element_blank(), panel.grid.minor = element_blank())

Loading environmetnal data

Next, we need to load some environmental data to use as predictors in our model.

# Load environmental predictors
# Monthly depth (static), SST (contemporaneous), and CHL (contemporaneous) for summer (jul-sep)
env_dat <- raster::brick("env_stack.tif")

# Set extent
extent(env_dat) <- extent(-82.92685,
                          -55.67632,
                          22.56053,
                          48.26303)

# Add names to raster stack -- were not saved in original tif -- workaround
vars <- c("Depth", "SST", "CHL")
years <- 2005:2015
names(env_dat) <- paste(rep(vars, times = 11), rep(years, each = 3), sep = ".")

env_dat
## class      : RasterBrick 
## dimensions : 575, 491, 282325, 33  (nrow, ncol, ncell, nlayers)
## resolution : 0.05550006, 0.0447  (x, y)
## extent     : -82.92685, -55.67632, 22.56053, 48.26303  (xmin, xmax, ymin, ymax)
## crs        : +proj=longlat +datum=WGS84 +no_defs 
## source     : env_stack.tif 
## names      :  Depth.2005,    SST.2005,    CHL.2005,  Depth.2006,    SST.2006,    CHL.2006,  Depth.2007,    SST.2007,    CHL.2007,  Depth.2008,    SST.2008,    CHL.2008,  Depth.2009,    SST.2009,    CHL.2009, ... 
## min values :  1.16924000,  9.86632633,  0.04109016,  1.16924000, 13.36531162,  0.04563540,  1.16924000, 11.37914085,  0.04466756,  1.16924000, 12.12702751,  0.03868118,  1.16924000, 11.19710350,  0.04641888, ... 
## max values :  5309.75977,    30.20734,    25.74244,  5309.75977,    29.67356,    27.01186,  5309.75977,    30.73302,    25.57668,  5309.75977,    29.92080,    23.06375,  5309.75977,    30.33197,    37.78852, ...
# Crop to bounding box
env_dat <- crop(env_dat, extent(BB))

# Plot first depth layer
plot(env_dat$Depth.2005)

# Plot first SST layer
plot(env_dat$SST.2005)

# Plot first CHL layer
plot(env_dat$CHL.2005)

### Format model data

An important thing to consider when producing SDMs is what sort of response you want to model. For example, SDMs can model presence-only, presence/absence, and abundance data. Approaches can be extended further to model things like density, but that requires the survey design to meet certain assumptions.

The right whale data from OBIS does have an individual count column, but the data are from many different sources, and likely of varying quality. Due to these variations in the data, a presence-only or presence/absence approach would make the most sense. Presence-only limits the types of models that can be used (e.g., MaxEnt is a good option), whereas a presence/absence approach is more readily adaptable to different model types. Another option is to create “pseudo-absences,” which are essentially fake absence data sampled from background points on the environmental data grid. Biomod2 has the ability to produce pseudo-absences, allowing us to produce models using algorithms beyond MaxEnt.

The next step is to format our data for the biomod2 package with a pseudo-absence generation approach.

# Add presence/absence column to right whale data -- binary
RightWhale <- RightWhale |>
  dplyr::mutate(pa = 1) # we will generate pseudo absences later

# Isolate binary presence/absence data - training years 2005-2010
trainingPA <- RightWhale |>
  dplyr::filter(year < 2011) |>
  dplyr::select(pa)

# Isolate presence/absence coordinates
trainingXY <- RightWhale |>
  dplyr::filter(year < 2011) |>
  dplyr::select(lon, lat)

We also want to set some data aside for model evaluation. For this, we will use 2011-2014. Biomod2 will generate pseudo-absences for the training data, but I am not sure if it does that for the evaluation data. So here is a workaround sampling from the boundaries of the environmental data for background points.

# Select background points and filter out NAs
# Ideally absence data is available
background <- as.data.frame(env_dat$Depth.2005, xy = TRUE) |> # pull locations from environmental data grid
  dplyr::rename(lon = x, lat = y) |>
  sample_n(30) |>
  dplyr::mutate(pa = 0, year = sample(c(2005, 2011), n(), replace = TRUE)) |>
  dplyr::filter(!is.na(Depth.2005)) |>
  dplyr::select(lon, lat, pa, year)

# Isolate binary presence/absence data - test years 2011-2014
evalPA <- RightWhale |>
  rbind(background) |> # attach background points
  dplyr::filter(year >= 2011 & year < 2015) |>
  dplyr::select(pa)
# Isolate presence/absence coordinates
evalXY <- RightWhale |>
  rbind(background) |> # attach background points
  dplyr::filter(year >= 2011 & year < 2015) |>
  dplyr::select(lon, lat)

We need to subset the environmental data raster to select only layers before 2010. For simplicity, we will take a climatalogical approach to the model by using the average summer SST and CHL values across the training time period. This is definitely not best practice.

# Create climatology of environmental training data for summer 2005-2010
# There is definitely a better way to do this
env_train <- raster::subset(env_dat, c(grep('2005', names(env_dat)),
                                        grep('2007', names(env_dat)),
                                        grep('2008', names(env_dat)),
                                        grep('2009', names(env_dat)),
                                        grep('2010', names(env_dat))))
sst <- raster::subset(env_train, grep('SST', names(env_train))) |>
  mean(na.rm = TRUE)
chl <- raster::subset(env_train, grep('CHL', names(env_train))) |>
  mean(na.rm = TRUE)
depth <- env_train$Depth.2005 # static variable

env_train <- stack(depth, sst, chl)
names(env_train) <- vars

Do the same for the evaluation environmental data

# Create climatology of environmental testing data for July 2011-2014
env_eval <- raster::subset(env_dat, c(grep('2011', names(env_dat)),
                                        grep('2012', names(env_dat)),
                                        grep('2013', names(env_dat)),
                                        grep('2014', names(env_dat))))
sst.eval <- raster::subset(env_eval, grep('SST', names(env_eval))) |>
  mean(na.rm = TRUE)
chl.eval <- raster::subset(env_eval, grep('CHL', names(env_eval))) |>
  mean(na.rm = TRUE)
depth.eval <- env_eval$Depth.2011 # static variable

env_eval <- stack(depth.eval, sst.eval, chl.eval)
names(env_eval) <- vars

Now that our data are ready to go, we will input them to the biomod2 data formatting function. We will try three different models: generalized linear models (GLMs), generalized additive models (GAMs), and random forests (RFs).

For a complete list of model options available in biomod2 and dependencies see model options vignette.

# Specify models to run
modelFormulas <- c('GLM', 'GAM', 'RF')

# Format data for use in Biomod2 modelling function & generate random pseudo-absences
biomodData <- BIOMOD_FormatingData(resp.var = trainingPA,
                                   expl.var = env_train,
                                   resp.xy = trainingXY,
                                   eval.resp.var = evalPA,
                                   eval.resp.xy = evalXY,
                                   eval.expl.var = env_eval,
                                   PA.nb.rep = 4, 
                                   PA.nb.absences = 200,
                                   PA.strategy = "random",
                                   resp.name = "RightWhale",
                                   filter.raster = TRUE)
## 
## -=-=-=-=-=-=-=-=-=-=-=-=-= RightWhale Data Formating -=-=-=-=-=-=-=-=-=-=-=-=-=
## 
##  !!! Some data are located in the same raster cell. 
##           Only the first data in each cell will be kept as `filter.raster = TRUE`.
## 
## Checking Pseudo-absence selection arguments...
## 
##    > random pseudo absences selection
##    > Pseudo absences are selected in explanatory variables
## 
##  !!! Some data are located in the same raster cell. 
##           Only the first data in each cell will be kept as `filter.raster = TRUE`.
##       ! No data has been set aside for modeling evaluation
## -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Done -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
biomodData
## 
## -=-=-=-=-=-=-=-=-=-=-=-=-=-= BIOMOD.formated.data -=-=-=-=-=-=-=-=-=-=-=-=-=-=
## 
## dir.name =  .
## 
## sp.name =  RightWhale
## 
##   87 presences,  0 true absences and  795 undefined points in dataset
## 
## 
##   3 explanatory variables
## 
##      Depth               SST             CHL          
##  Min.   :   2.583   Min.   :11.65   Min.   : 0.04995  
##  1st Qu.:  88.737   1st Qu.:16.47   1st Qu.: 0.15971  
##  Median : 206.703   Median :20.22   Median : 0.40891  
##  Mean   :1445.225   Mean   :20.65   Mean   : 0.92844  
##  3rd Qu.:3005.230   3rd Qu.:25.17   3rd Qu.: 0.75132  
##  Max.   :4783.216   Max.   :28.49   Max.   :25.38794  
## 
## 
## Evaluation data :
## 
##   26 presences,  4 true absences and  0 undefined points in dataset
## 
## 
## 
##      Depth              SST             CHL        
##  Min.   :  43.43   Min.   :12.69   Min.   :0.1198  
##  1st Qu.: 147.68   1st Qu.:16.62   1st Qu.:0.5835  
##  Median : 175.19   Median :16.88   Median :0.6138  
##  Mean   : 397.74   Mean   :17.91   Mean   :0.8196  
##  3rd Qu.: 221.80   3rd Qu.:19.01   3rd Qu.:0.6374  
##  Max.   :4354.47   Max.   :25.50   Max.   :7.2483  
## 
## 
##  4 Pseudo Absences dataset available ( PA1, PA2, PA3, PA4 ) with  
## 200 (PA1, PA2, PA3, PA4) pseudo absences
## 
## -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

Build models

We are now ready for the modeling stage.

# Build the models
modelOut <- BIOMOD_Modeling(bm.format = biomodData,
                            modeling.id = "RightWhale",
                            models = c("GLM", "GAM", "RF"),
                            CV.nb.rep = 5, # 5-fold cross validation
                            data.split.perc = 70, # 70%/30% training/testing data split
                            prevalence = 0.5,
                            var.import = 5,
                            metric.eval = c('ROC', 'TSS', 'KAPPA'),
                            scale.models = FALSE,
                            do.progress = TRUE)
## Warning in .bm_ModelingOptions.check.args(data.type = data.type, models =
## models, : Only one GAM model can be activated. 'GAM.mgcv.gam' has been set
## (other available : 'GAM.gam.gam' or 'GAM.mgcv.bam')
## Warning: executing %dopar% sequentially: no parallel backend registered
## Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred
## Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred
## Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred
## Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred
## Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred
## Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred
## Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred
## Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred
## Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred
## Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred
## Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred
## Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred
## Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred
## Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred
## Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred
## Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred
## Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred
## Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred
## Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred
## Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred
## Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred
## Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred
## Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred
## Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred
## Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred

Model evaluations

Evaluations for each model formula, evaluation, and estimate of pseudo-absences is available in the output model object.

# Extract model evaluations
modelEvals <- get_evaluations(obj = modelOut)
modelEvals
##                         full.name      PA    run algo metric.eval cutoff
## 1         RightWhale_PA1_RUN1_GLM     PA1   RUN1  GLM         ROC  422.0
## 2         RightWhale_PA1_RUN1_GLM     PA1   RUN1  GLM         TSS  423.0
## 3         RightWhale_PA1_RUN1_GLM     PA1   RUN1  GLM       KAPPA  526.0
## 4         RightWhale_PA1_RUN1_GAM     PA1   RUN1  GAM         ROC  552.5
## 5         RightWhale_PA1_RUN1_GAM     PA1   RUN1  GAM         TSS  553.5
## 6         RightWhale_PA1_RUN1_GAM     PA1   RUN1  GAM       KAPPA  553.5
## 7          RightWhale_PA1_RUN1_RF     PA1   RUN1   RF         ROC  527.0
## 8          RightWhale_PA1_RUN1_RF     PA1   RUN1   RF         TSS  525.0
## 9          RightWhale_PA1_RUN1_RF     PA1   RUN1   RF       KAPPA  525.0
## 10        RightWhale_PA1_RUN2_GLM     PA1   RUN2  GLM         ROC  470.5
## 11        RightWhale_PA1_RUN2_GLM     PA1   RUN2  GLM         TSS  470.0
## 12        RightWhale_PA1_RUN2_GLM     PA1   RUN2  GLM       KAPPA  759.0
## 13        RightWhale_PA1_RUN2_GAM     PA1   RUN2  GAM         ROC  525.0
## 14        RightWhale_PA1_RUN2_GAM     PA1   RUN2  GAM         TSS  525.5
## 15        RightWhale_PA1_RUN2_GAM     PA1   RUN2  GAM       KAPPA  593.0
## 16         RightWhale_PA1_RUN2_RF     PA1   RUN2   RF         ROC  506.0
## 17         RightWhale_PA1_RUN2_RF     PA1   RUN2   RF         TSS  505.0
## 18         RightWhale_PA1_RUN2_RF     PA1   RUN2   RF       KAPPA  505.0
## 19        RightWhale_PA1_RUN3_GLM     PA1   RUN3  GLM         ROC  367.5
## 20        RightWhale_PA1_RUN3_GLM     PA1   RUN3  GLM         TSS  364.0
## 21        RightWhale_PA1_RUN3_GLM     PA1   RUN3  GLM       KAPPA  620.0
## 22        RightWhale_PA1_RUN3_GAM     PA1   RUN3  GAM         ROC  577.5
## 23        RightWhale_PA1_RUN3_GAM     PA1   RUN3  GAM         TSS  575.5
## 24        RightWhale_PA1_RUN3_GAM     PA1   RUN3  GAM       KAPPA  575.5
## 25         RightWhale_PA1_RUN3_RF     PA1   RUN3   RF         ROC  512.0
## 26         RightWhale_PA1_RUN3_RF     PA1   RUN3   RF         TSS  510.0
## 27         RightWhale_PA1_RUN3_RF     PA1   RUN3   RF       KAPPA  510.0
## 28        RightWhale_PA1_RUN4_GLM     PA1   RUN4  GLM         ROC  433.5
## 29        RightWhale_PA1_RUN4_GLM     PA1   RUN4  GLM         TSS  431.0
## 30        RightWhale_PA1_RUN4_GLM     PA1   RUN4  GLM       KAPPA  713.0
## 31        RightWhale_PA1_RUN4_GAM     PA1   RUN4  GAM         ROC  582.0
## 32        RightWhale_PA1_RUN4_GAM     PA1   RUN4  GAM         TSS  568.0
## 33        RightWhale_PA1_RUN4_GAM     PA1   RUN4  GAM       KAPPA  568.0
## 34         RightWhale_PA1_RUN4_RF     PA1   RUN4   RF         ROC  480.0
## 35         RightWhale_PA1_RUN4_RF     PA1   RUN4   RF         TSS  475.0
## 36         RightWhale_PA1_RUN4_RF     PA1   RUN4   RF       KAPPA  475.0
## 37        RightWhale_PA1_RUN5_GLM     PA1   RUN5  GLM         ROC  666.5
## 38        RightWhale_PA1_RUN5_GLM     PA1   RUN5  GLM         TSS  366.0
## 39        RightWhale_PA1_RUN5_GLM     PA1   RUN5  GLM       KAPPA  712.0
## 40        RightWhale_PA1_RUN5_GAM     PA1   RUN5  GAM         ROC  586.5
## 41        RightWhale_PA1_RUN5_GAM     PA1   RUN5  GAM         TSS  565.5
## 42        RightWhale_PA1_RUN5_GAM     PA1   RUN5  GAM       KAPPA  565.5
## 43         RightWhale_PA1_RUN5_RF     PA1   RUN5   RF         ROC  497.0
## 44         RightWhale_PA1_RUN5_RF     PA1   RUN5   RF         TSS  500.0
## 45         RightWhale_PA1_RUN5_RF     PA1   RUN5   RF       KAPPA  500.0
## 46      RightWhale_PA1_allRun_GLM     PA1 allRun  GLM         ROC  364.0
## 47      RightWhale_PA1_allRun_GLM     PA1 allRun  GLM         TSS  363.0
## 48      RightWhale_PA1_allRun_GLM     PA1 allRun  GLM       KAPPA  716.0
## 49      RightWhale_PA1_allRun_GAM     PA1 allRun  GAM         ROC  572.0
## 50      RightWhale_PA1_allRun_GAM     PA1 allRun  GAM         TSS  574.0
## 51      RightWhale_PA1_allRun_GAM     PA1 allRun  GAM       KAPPA  583.0
## 52       RightWhale_PA1_allRun_RF     PA1 allRun   RF         ROC  508.0
## 53       RightWhale_PA1_allRun_RF     PA1 allRun   RF         TSS  510.0
## 54       RightWhale_PA1_allRun_RF     PA1 allRun   RF       KAPPA  510.0
## 55        RightWhale_PA2_RUN1_GLM     PA2   RUN1  GLM         ROC  496.5
## 56        RightWhale_PA2_RUN1_GLM     PA2   RUN1  GLM         TSS  496.0
## 57        RightWhale_PA2_RUN1_GLM     PA2   RUN1  GLM       KAPPA  496.0
## 58        RightWhale_PA2_RUN1_GAM     PA2   RUN1  GAM         ROC  543.5
## 59        RightWhale_PA2_RUN1_GAM     PA2   RUN1  GAM         TSS  537.0
## 60        RightWhale_PA2_RUN1_GAM     PA2   RUN1  GAM       KAPPA  537.0
## 61         RightWhale_PA2_RUN1_RF     PA2   RUN1   RF         ROC  491.0
## 62         RightWhale_PA2_RUN1_RF     PA2   RUN1   RF         TSS  493.0
## 63         RightWhale_PA2_RUN1_RF     PA2   RUN1   RF       KAPPA  493.0
## 64        RightWhale_PA2_RUN2_GLM     PA2   RUN2  GLM         ROC  356.5
## 65        RightWhale_PA2_RUN2_GLM     PA2   RUN2  GLM         TSS  355.0
## 66        RightWhale_PA2_RUN2_GLM     PA2   RUN2  GLM       KAPPA  458.0
## 67        RightWhale_PA2_RUN2_GAM     PA2   RUN2  GAM         ROC  558.5
## 68        RightWhale_PA2_RUN2_GAM     PA2   RUN2  GAM         TSS  556.0
## 69        RightWhale_PA2_RUN2_GAM     PA2   RUN2  GAM       KAPPA  556.0
## 70         RightWhale_PA2_RUN2_RF     PA2   RUN2   RF         ROC  532.0
## 71         RightWhale_PA2_RUN2_RF     PA2   RUN2   RF         TSS  530.0
## 72         RightWhale_PA2_RUN2_RF     PA2   RUN2   RF       KAPPA  530.0
## 73        RightWhale_PA2_RUN3_GLM     PA2   RUN3  GLM         ROC  479.0
## 74        RightWhale_PA2_RUN3_GLM     PA2   RUN3  GLM         TSS  547.0
## 75        RightWhale_PA2_RUN3_GLM     PA2   RUN3  GLM       KAPPA  686.0
## 76        RightWhale_PA2_RUN3_GAM     PA2   RUN3  GAM         ROC  543.5
## 77        RightWhale_PA2_RUN3_GAM     PA2   RUN3  GAM         TSS  521.0
## 78        RightWhale_PA2_RUN3_GAM     PA2   RUN3  GAM       KAPPA  544.0
## 79         RightWhale_PA2_RUN3_RF     PA2   RUN3   RF         ROC  504.0
## 80         RightWhale_PA2_RUN3_RF     PA2   RUN3   RF         TSS  505.0
## 81         RightWhale_PA2_RUN3_RF     PA2   RUN3   RF       KAPPA  505.0
## 82        RightWhale_PA2_RUN4_GLM     PA2   RUN4  GLM         ROC  392.5
## 83        RightWhale_PA2_RUN4_GLM     PA2   RUN4  GLM         TSS  395.0
## 84        RightWhale_PA2_RUN4_GLM     PA2   RUN4  GLM       KAPPA  395.0
## 85        RightWhale_PA2_RUN4_GAM     PA2   RUN4  GAM         ROC  508.5
## 86        RightWhale_PA2_RUN4_GAM     PA2   RUN4  GAM         TSS  507.0
## 87        RightWhale_PA2_RUN4_GAM     PA2   RUN4  GAM       KAPPA  507.0
## 88         RightWhale_PA2_RUN4_RF     PA2   RUN4   RF         ROC  538.0
## 89         RightWhale_PA2_RUN4_RF     PA2   RUN4   RF         TSS  540.0
## 90         RightWhale_PA2_RUN4_RF     PA2   RUN4   RF       KAPPA  540.0
## 91        RightWhale_PA2_RUN5_GLM     PA2   RUN5  GLM         ROC  434.5
## 92        RightWhale_PA2_RUN5_GLM     PA2   RUN5  GLM         TSS  435.0
## 93        RightWhale_PA2_RUN5_GLM     PA2   RUN5  GLM       KAPPA  554.0
## 94        RightWhale_PA2_RUN5_GAM     PA2   RUN5  GAM         ROC  504.0
## 95        RightWhale_PA2_RUN5_GAM     PA2   RUN5  GAM         TSS  505.5
## 96        RightWhale_PA2_RUN5_GAM     PA2   RUN5  GAM       KAPPA  505.5
## 97         RightWhale_PA2_RUN5_RF     PA2   RUN5   RF         ROC  507.0
## 98         RightWhale_PA2_RUN5_RF     PA2   RUN5   RF         TSS  509.0
## 99         RightWhale_PA2_RUN5_RF     PA2   RUN5   RF       KAPPA  509.0
## 100     RightWhale_PA2_allRun_GLM     PA2 allRun  GLM         ROC  547.5
## 101     RightWhale_PA2_allRun_GLM     PA2 allRun  GLM         TSS  540.0
## 102     RightWhale_PA2_allRun_GLM     PA2 allRun  GLM       KAPPA  540.0
## 103     RightWhale_PA2_allRun_GAM     PA2 allRun  GAM         ROC  539.0
## 104     RightWhale_PA2_allRun_GAM     PA2 allRun  GAM         TSS  539.0
## 105     RightWhale_PA2_allRun_GAM     PA2 allRun  GAM       KAPPA  539.0
## 106      RightWhale_PA2_allRun_RF     PA2 allRun   RF         ROC  512.0
## 107      RightWhale_PA2_allRun_RF     PA2 allRun   RF         TSS  514.0
## 108      RightWhale_PA2_allRun_RF     PA2 allRun   RF       KAPPA  514.0
## 109       RightWhale_PA3_RUN1_GLM     PA3   RUN1  GLM         ROC  490.5
## 110       RightWhale_PA3_RUN1_GLM     PA3   RUN1  GLM         TSS  491.0
## 111       RightWhale_PA3_RUN1_GLM     PA3   RUN1  GLM       KAPPA  491.0
## 112       RightWhale_PA3_RUN1_GAM     PA3   RUN1  GAM         ROC  515.5
## 113       RightWhale_PA3_RUN1_GAM     PA3   RUN1  GAM         TSS  555.0
## 114       RightWhale_PA3_RUN1_GAM     PA3   RUN1  GAM       KAPPA  555.0
## 115        RightWhale_PA3_RUN1_RF     PA3   RUN1   RF         ROC  516.0
## 116        RightWhale_PA3_RUN1_RF     PA3   RUN1   RF         TSS  515.0
## 117        RightWhale_PA3_RUN1_RF     PA3   RUN1   RF       KAPPA  515.0
## 118       RightWhale_PA3_RUN2_GLM     PA3   RUN2  GLM         ROC  519.0
## 119       RightWhale_PA3_RUN2_GLM     PA3   RUN2  GLM         TSS  518.0
## 120       RightWhale_PA3_RUN2_GLM     PA3   RUN2  GLM       KAPPA  518.0
## 121       RightWhale_PA3_RUN2_GAM     PA3   RUN2  GAM         ROC  530.0
## 122       RightWhale_PA3_RUN2_GAM     PA3   RUN2  GAM         TSS  528.5
## 123       RightWhale_PA3_RUN2_GAM     PA3   RUN2  GAM       KAPPA  578.0
## 124        RightWhale_PA3_RUN2_RF     PA3   RUN2   RF         ROC  514.0
## 125        RightWhale_PA3_RUN2_RF     PA3   RUN2   RF         TSS  510.0
## 126        RightWhale_PA3_RUN2_RF     PA3   RUN2   RF       KAPPA  510.0
## 127       RightWhale_PA3_RUN3_GLM     PA3   RUN3  GLM         ROC  339.5
## 128       RightWhale_PA3_RUN3_GLM     PA3   RUN3  GLM         TSS  750.0
## 129       RightWhale_PA3_RUN3_GLM     PA3   RUN3  GLM       KAPPA  750.0
## 130       RightWhale_PA3_RUN3_GAM     PA3   RUN3  GAM         ROC  560.5
## 131       RightWhale_PA3_RUN3_GAM     PA3   RUN3  GAM         TSS  560.5
## 132       RightWhale_PA3_RUN3_GAM     PA3   RUN3  GAM       KAPPA  560.5
## 133        RightWhale_PA3_RUN3_RF     PA3   RUN3   RF         ROC  509.0
## 134        RightWhale_PA3_RUN3_RF     PA3   RUN3   RF         TSS  507.0
## 135        RightWhale_PA3_RUN3_RF     PA3   RUN3   RF       KAPPA  507.0
## 136       RightWhale_PA3_RUN4_GLM     PA3   RUN4  GLM         ROC  561.0
## 137       RightWhale_PA3_RUN4_GLM     PA3   RUN4  GLM         TSS  562.0
## 138       RightWhale_PA3_RUN4_GLM     PA3   RUN4  GLM       KAPPA  562.0
## 139       RightWhale_PA3_RUN4_GAM     PA3   RUN4  GAM         ROC  534.5
## 140       RightWhale_PA3_RUN4_GAM     PA3   RUN4  GAM         TSS  531.5
## 141       RightWhale_PA3_RUN4_GAM     PA3   RUN4  GAM       KAPPA  531.5
## 142        RightWhale_PA3_RUN4_RF     PA3   RUN4   RF         ROC  531.0
## 143        RightWhale_PA3_RUN4_RF     PA3   RUN4   RF         TSS  528.0
## 144        RightWhale_PA3_RUN4_RF     PA3   RUN4   RF       KAPPA  528.0
## 145       RightWhale_PA3_RUN5_GLM     PA3   RUN5  GLM         ROC  545.0
## 146       RightWhale_PA3_RUN5_GLM     PA3   RUN5  GLM         TSS  545.0
## 147       RightWhale_PA3_RUN5_GLM     PA3   RUN5  GLM       KAPPA  545.0
## 148       RightWhale_PA3_RUN5_GAM     PA3   RUN5  GAM         ROC  607.5
## 149       RightWhale_PA3_RUN5_GAM     PA3   RUN5  GAM         TSS  587.0
## 150       RightWhale_PA3_RUN5_GAM     PA3   RUN5  GAM       KAPPA  587.0
## 151        RightWhale_PA3_RUN5_RF     PA3   RUN5   RF         ROC  515.0
## 152        RightWhale_PA3_RUN5_RF     PA3   RUN5   RF         TSS  515.0
## 153        RightWhale_PA3_RUN5_RF     PA3   RUN5   RF       KAPPA  515.0
## 154     RightWhale_PA3_allRun_GLM     PA3 allRun  GLM         ROC  527.5
## 155     RightWhale_PA3_allRun_GLM     PA3 allRun  GLM         TSS  525.0
## 156     RightWhale_PA3_allRun_GLM     PA3 allRun  GLM       KAPPA  525.0
## 157     RightWhale_PA3_allRun_GAM     PA3 allRun  GAM         ROC  554.5
## 158     RightWhale_PA3_allRun_GAM     PA3 allRun  GAM         TSS  551.5
## 159     RightWhale_PA3_allRun_GAM     PA3 allRun  GAM       KAPPA  551.5
## 160      RightWhale_PA3_allRun_RF     PA3 allRun   RF         ROC  526.0
## 161      RightWhale_PA3_allRun_RF     PA3 allRun   RF         TSS  529.0
## 162      RightWhale_PA3_allRun_RF     PA3 allRun   RF       KAPPA  529.0
## 163       RightWhale_PA4_RUN1_GLM     PA4   RUN1  GLM         ROC  343.5
## 164       RightWhale_PA4_RUN1_GLM     PA4   RUN1  GLM         TSS  341.0
## 165       RightWhale_PA4_RUN1_GLM     PA4   RUN1  GLM       KAPPA  636.0
## 166       RightWhale_PA4_RUN1_GAM     PA4   RUN1  GAM         ROC  584.0
## 167       RightWhale_PA4_RUN1_GAM     PA4   RUN1  GAM         TSS  582.0
## 168       RightWhale_PA4_RUN1_GAM     PA4   RUN1  GAM       KAPPA  582.0
## 169        RightWhale_PA4_RUN1_RF     PA4   RUN1   RF         ROC  505.0
## 170        RightWhale_PA4_RUN1_RF     PA4   RUN1   RF         TSS  505.0
## 171        RightWhale_PA4_RUN1_RF     PA4   RUN1   RF       KAPPA  505.0
## 172       RightWhale_PA4_RUN2_GLM     PA4   RUN2  GLM         ROC  375.5
## 173       RightWhale_PA4_RUN2_GLM     PA4   RUN2  GLM         TSS  371.0
## 174       RightWhale_PA4_RUN2_GLM     PA4   RUN2  GLM       KAPPA  757.0
## 175       RightWhale_PA4_RUN2_GAM     PA4   RUN2  GAM         ROC  553.0
## 176       RightWhale_PA4_RUN2_GAM     PA4   RUN2  GAM         TSS  553.0
## 177       RightWhale_PA4_RUN2_GAM     PA4   RUN2  GAM       KAPPA  553.0
## 178        RightWhale_PA4_RUN2_RF     PA4   RUN2   RF         ROC  502.0
## 179        RightWhale_PA4_RUN2_RF     PA4   RUN2   RF         TSS  505.0
## 180        RightWhale_PA4_RUN2_RF     PA4   RUN2   RF       KAPPA  505.0
## 181       RightWhale_PA4_RUN3_GLM     PA4   RUN3  GLM         ROC  395.0
## 182       RightWhale_PA4_RUN3_GLM     PA4   RUN3  GLM         TSS  395.0
## 183       RightWhale_PA4_RUN3_GLM     PA4   RUN3  GLM       KAPPA  395.0
## 184       RightWhale_PA4_RUN3_GAM     PA4   RUN3  GAM         ROC  591.0
## 185       RightWhale_PA4_RUN3_GAM     PA4   RUN3  GAM         TSS  589.0
## 186       RightWhale_PA4_RUN3_GAM     PA4   RUN3  GAM       KAPPA  589.0
## 187        RightWhale_PA4_RUN3_RF     PA4   RUN3   RF         ROC  535.0
## 188        RightWhale_PA4_RUN3_RF     PA4   RUN3   RF         TSS  530.0
## 189        RightWhale_PA4_RUN3_RF     PA4   RUN3   RF       KAPPA  530.0
## 190       RightWhale_PA4_RUN4_GLM     PA4   RUN4  GLM         ROC  394.5
## 191       RightWhale_PA4_RUN4_GLM     PA4   RUN4  GLM         TSS  395.0
## 192       RightWhale_PA4_RUN4_GLM     PA4   RUN4  GLM       KAPPA  704.0
## 193       RightWhale_PA4_RUN4_GAM     PA4   RUN4  GAM         ROC  578.0
## 194       RightWhale_PA4_RUN4_GAM     PA4   RUN4  GAM         TSS  567.0
## 195       RightWhale_PA4_RUN4_GAM     PA4   RUN4  GAM       KAPPA  567.0
## 196        RightWhale_PA4_RUN4_RF     PA4   RUN4   RF         ROC  533.0
## 197        RightWhale_PA4_RUN4_RF     PA4   RUN4   RF         TSS  530.0
## 198        RightWhale_PA4_RUN4_RF     PA4   RUN4   RF       KAPPA  530.0
## 199       RightWhale_PA4_RUN5_GLM     PA4   RUN5  GLM         ROC  419.5
## 200       RightWhale_PA4_RUN5_GLM     PA4   RUN5  GLM         TSS  421.0
## 201       RightWhale_PA4_RUN5_GLM     PA4   RUN5  GLM       KAPPA  697.0
## 202       RightWhale_PA4_RUN5_GAM     PA4   RUN5  GAM         ROC  510.0
## 203       RightWhale_PA4_RUN5_GAM     PA4   RUN5  GAM         TSS  508.0
## 204       RightWhale_PA4_RUN5_GAM     PA4   RUN5  GAM       KAPPA  550.0
## 205        RightWhale_PA4_RUN5_RF     PA4   RUN5   RF         ROC  517.0
## 206        RightWhale_PA4_RUN5_RF     PA4   RUN5   RF         TSS  515.0
## 207        RightWhale_PA4_RUN5_RF     PA4   RUN5   RF       KAPPA  515.0
## 208     RightWhale_PA4_allRun_GLM     PA4 allRun  GLM         ROC  395.5
## 209     RightWhale_PA4_allRun_GLM     PA4 allRun  GLM         TSS  395.0
## 210     RightWhale_PA4_allRun_GLM     PA4 allRun  GLM       KAPPA  395.0
## 211     RightWhale_PA4_allRun_GAM     PA4 allRun  GAM         ROC  566.5
## 212     RightWhale_PA4_allRun_GAM     PA4 allRun  GAM         TSS  563.0
## 213     RightWhale_PA4_allRun_GAM     PA4 allRun  GAM       KAPPA  563.0
## 214      RightWhale_PA4_allRun_RF     PA4 allRun   RF         ROC  544.0
## 215      RightWhale_PA4_allRun_RF     PA4 allRun   RF         TSS  540.0
## 216      RightWhale_PA4_allRun_RF     PA4 allRun   RF       KAPPA  540.0
## 217 RightWhale_allData_allRun_GLM allData allRun  GLM         ROC   96.5
## 218 RightWhale_allData_allRun_GLM allData allRun  GLM         TSS   95.0
## 219 RightWhale_allData_allRun_GLM allData allRun  GLM       KAPPA  273.0
## 220 RightWhale_allData_allRun_GAM allData allRun  GAM         ROC  151.0
## 221 RightWhale_allData_allRun_GAM allData allRun  GAM         TSS  151.0
## 222 RightWhale_allData_allRun_GAM allData allRun  GAM       KAPPA  151.0
## 223  RightWhale_allData_allRun_RF allData allRun   RF         ROC  503.0
## 224  RightWhale_allData_allRun_RF allData allRun   RF         TSS  502.0
## 225  RightWhale_allData_allRun_RF allData allRun   RF       KAPPA  502.0
##     sensitivity specificity calibration validation evaluation
## 1        98.361      83.571       0.946      0.970      0.971
## 2        98.361      83.571       0.819      0.850      0.962
## 3        95.082      86.429       0.759      0.774      0.870
## 4        96.721      70.714       0.815      0.830      0.505
## 5        96.721      70.714       0.674      0.717      0.462
## 6        96.721      70.714       0.571      0.605      0.516
## 7       100.000     100.000       1.000      0.994      0.981
## 8       100.000     100.000       1.000      0.950      0.962
## 9       100.000     100.000       1.000      0.920      0.870
## 10       95.082      88.571       0.969      0.935      0.971
## 11       95.082      88.571       0.837      0.663      0.962
## 12       86.885      95.000       0.823      0.687      0.870
## 13       98.361      72.143       0.825      0.807      0.505
## 14       98.361      72.143       0.705      0.612      0.462
## 15       95.082      75.000       0.610      0.512      0.516
## 16      100.000     100.000       1.000      0.983      0.971
## 17      100.000     100.000       1.000      0.890      0.962
## 18      100.000     100.000       1.000      0.890      0.870
## 19      100.000      85.714       0.958      0.943      0.971
## 20      100.000      85.714       0.857      0.773      0.962
## 21       93.443      90.000       0.798      0.864      0.870
## 22      100.000      73.571       0.825      0.746      0.582
## 23      100.000      73.571       0.736      0.606      0.462
## 24      100.000      73.571       0.628      0.512      0.516
## 25      100.000     100.000       1.000      0.981      0.971
## 26      100.000     100.000       1.000      0.851      0.962
## 27      100.000     100.000       1.000      0.861      0.870
## 28       98.361      85.000       0.958      0.960      0.971
## 29       98.361      85.000       0.834      0.778      0.962
## 30       86.885      92.857       0.790      0.760      0.870
## 31       98.361      73.571       0.827      0.798      0.505
## 32       98.361      72.857       0.712      0.645      0.462
## 33       98.361      72.857       0.608      0.539      0.516
## 34      100.000     100.000       1.000      0.978      0.981
## 35      100.000     100.000       1.000      0.912      0.962
## 36      100.000     100.000       1.000      0.892      0.870
## 37       91.803      92.143       0.964      0.946      0.971
## 38       98.361      85.000       0.834      0.762      0.962
## 39       90.164      92.857       0.815      0.724      0.870
## 40       98.361      72.857       0.833      0.782      0.505
## 41       98.361      72.143       0.705      0.628      0.462
## 42       98.361      72.143       0.600      0.520      0.516
## 43      100.000     100.000       1.000      0.976      0.971
## 44      100.000     100.000       1.000      0.890      0.923
## 45      100.000     100.000       1.000      0.890      0.762
## 46       98.851      83.500       0.958         NA      0.971
## 47       98.851      83.500       0.824         NA      0.962
## 48       87.356      93.000       0.796         NA      0.870
## 49       97.701      71.000       0.819         NA      0.505
## 50       97.701      71.000       0.687         NA      0.462
## 51       96.552      71.500       0.585         NA      0.516
## 52      100.000     100.000       1.000         NA      0.971
## 53      100.000     100.000       1.000         NA      0.962
## 54      100.000     100.000       1.000         NA      0.870
## 55       98.361      87.143       0.930      0.945      0.971
## 56       98.361      87.143       0.855      0.845      0.962
## 57       98.361      87.143       0.793      0.793      0.870
## 58      100.000      76.429       0.840      0.806      0.514
## 59      100.000      75.714       0.757      0.612      0.462
## 60      100.000      75.714       0.654      0.502      0.516
## 61      100.000     100.000       1.000      0.989      0.846
## 62      100.000     100.000       1.000      0.862      0.615
## 63      100.000     100.000       1.000      0.817      0.634
## 64       98.361      86.429       0.930      0.920      0.971
## 65       98.361      86.429       0.848      0.733      0.962
## 66       96.721      87.857       0.791      0.707      0.870
## 67      100.000      75.714       0.824      0.816      0.514
## 68      100.000      75.714       0.757      0.700      0.462
## 69      100.000      75.714       0.654      0.585      0.516
## 70      100.000     100.000       1.000      0.971      0.933
## 71      100.000     100.000       1.000      0.856      0.846
## 72      100.000     100.000       1.000      0.838      0.634
## 73       96.721      87.143       0.934      0.931      0.971
## 74       95.082      88.571       0.837      0.751      0.962
## 75       91.803      90.714       0.796      0.730      0.870
## 76       98.361      75.000       0.843      0.807      0.514
## 77       98.361      73.571       0.719      0.683      0.462
## 78       98.361      75.000       0.622      0.566      0.516
## 79      100.000     100.000       1.000      0.981      0.899
## 80      100.000     100.000       1.000      0.801      0.673
## 81      100.000     100.000       1.000      0.784      0.634
## 82      100.000      88.571       0.958      0.924      0.942
## 83      100.000      88.571       0.886      0.740      0.885
## 84      100.000      88.571       0.825      0.674      0.672
## 85      100.000      78.571       0.864      0.738      0.514
## 86      100.000      78.571       0.786      0.578      0.462
## 87      100.000      78.571       0.690      0.466      0.516
## 88      100.000     100.000       1.000      0.967      0.846
## 89      100.000     100.000       1.000      0.813      0.654
## 90      100.000     100.000       1.000      0.831      0.516
## 91       96.721      85.714       0.936      0.914      0.971
## 92       96.721      85.714       0.824      0.867      0.962
## 93       95.082      87.143       0.769      0.797      0.870
## 94       98.361      70.000       0.805      0.886      0.514
## 95       98.361      70.000       0.684      0.817      0.462
## 96       98.361      70.000       0.575      0.729      0.516
## 97      100.000     100.000       1.000      0.990      0.865
## 98      100.000     100.000       1.000      0.906      0.692
## 99      100.000     100.000       1.000      0.916      0.634
## 100      96.552      88.000       0.933         NA      0.971
## 101      96.552      87.500       0.841         NA      0.962
## 102      96.552      87.500       0.784         NA      0.870
## 103      98.851      74.000       0.828         NA      0.514
## 104      98.851      74.000       0.729         NA      0.462
## 105      98.851      74.000       0.625         NA      0.516
## 106     100.000     100.000       1.000         NA      0.841
## 107     100.000     100.000       1.000         NA      0.654
## 108     100.000     100.000       1.000         NA      0.516
## 109      98.361      89.286       0.950      0.905      0.971
## 110      98.361      89.286       0.876      0.795      0.962
## 111      98.361      89.286       0.823      0.724      0.870
## 112     100.000      78.571       0.848      0.751      0.519
## 113      98.361      80.000       0.784      0.683      0.462
## 114      98.361      80.000       0.697      0.566      0.516
## 115     100.000     100.000       1.000      0.950      0.966
## 116     100.000     100.000       1.000      0.785      0.923
## 117     100.000     100.000       1.000      0.760      0.762
## 118      96.721      87.857       0.945      0.923      0.971
## 119      96.721      87.857       0.846      0.850      0.962
## 120      96.721      87.857       0.791      0.774      0.870
## 121     100.000      76.429       0.829      0.796      0.514
## 122     100.000      76.429       0.764      0.717      0.462
## 123      98.361      77.857       0.669      0.597      0.516
## 124     100.000     100.000       1.000      0.928      0.962
## 125     100.000     100.000       1.000      0.768      0.923
## 126     100.000     100.000       1.000      0.736      0.762
## 127      98.361      85.714       0.954      0.942      0.971
## 128      91.803      92.143       0.839      0.571      0.962
## 129      91.803      92.143       0.817      0.597      0.870
## 130     100.000      71.429       0.787      0.903      0.505
## 131     100.000      71.429       0.714      0.812      0.462
## 132     100.000      71.429       0.603      0.746      0.516
## 133     100.000     100.000       1.000      0.957      0.971
## 134     100.000     100.000       1.000      0.703      0.923
## 135     100.000     100.000       1.000      0.718      0.762
## 136      96.721      86.429       0.936      0.936      0.971
## 137      96.721      86.429       0.831      0.845      0.962
## 138      96.721      86.429       0.771      0.793      0.870
## 139      98.361      75.000       0.826      0.826      0.514
## 140      98.361      75.000       0.734      0.800      0.462
## 141      98.361      75.000       0.634      0.707      0.516
## 142     100.000     100.000       1.000      0.956      0.962
## 143     100.000     100.000       1.000      0.703      0.923
## 144     100.000     100.000       1.000      0.718      0.762
## 145      96.721      86.429       0.935      0.954      0.971
## 146      96.721      86.429       0.831      0.900      0.962
## 147      96.721      86.429       0.771      0.845      0.870
## 148     100.000      78.571       0.829      0.787      0.514
## 149     100.000      77.857       0.779      0.683      0.462
## 150     100.000      77.857       0.681      0.566      0.516
## 151     100.000     100.000       1.000      0.989      0.962
## 152     100.000     100.000       1.000      0.933      0.923
## 153     100.000     100.000       1.000      0.894      0.762
## 154      96.552      88.000       0.943         NA      0.971
## 155      96.552      88.000       0.846         NA      0.962
## 156      96.552      88.000       0.792         NA      0.870
## 157      98.851      76.500       0.821         NA      0.514
## 158      98.851      76.500       0.754         NA      0.462
## 159      98.851      76.500       0.656         NA      0.516
## 160     100.000     100.000       1.000         NA      0.962
## 161     100.000     100.000       1.000         NA      0.923
## 162     100.000     100.000       1.000         NA      0.762
## 163      98.361      90.000       0.959      0.964      0.971
## 164      98.361      89.286       0.884      0.768      0.962
## 165      91.803      93.571       0.838      0.718      0.870
## 166      98.361      62.143       0.723      0.713      0.510
## 167      98.361      62.143       0.605      0.612      0.462
## 168      98.361      62.143       0.488      0.502      0.516
## 169     100.000     100.000       1.000      0.978      0.981
## 170     100.000     100.000       1.000      0.835      0.962
## 171     100.000     100.000       1.000      0.835      0.870
## 172      96.721      90.000       0.964      0.947      0.971
## 173      96.721      89.286       0.867      0.790      0.962
## 174      88.525      94.286       0.824      0.779      0.870
## 175      98.361      67.143       0.758      0.754      0.505
## 176      98.361      67.143       0.655      0.700      0.462
## 177      98.361      67.143       0.542      0.585      0.516
## 178     100.000     100.000       1.000      0.982      0.971
## 179     100.000     100.000       1.000      0.873      0.962
## 180     100.000     100.000       1.000      0.864      0.870
## 181      98.361      91.429       0.966      0.941      0.971
## 182      98.361      91.429       0.898      0.713      0.962
## 183      98.361      91.429       0.854      0.683      0.870
## 184      98.361      70.714       0.765      0.706      0.510
## 185      98.361      70.714       0.691      0.633      0.462
## 186      98.361      70.714       0.583      0.511      0.516
## 187     100.000     100.000       1.000      0.963      0.981
## 188     100.000     100.000       1.000      0.708      0.962
## 189     100.000     100.000       1.000      0.700      0.870
## 190      96.721      88.571       0.955      0.963      0.971
## 191      96.721      88.571       0.853      0.895      0.962
## 192      88.525      93.571       0.813      0.805      0.870
## 193      98.361      69.286       0.741      0.765      0.505
## 194      98.361      68.571       0.669      0.683      0.462
## 195      98.361      68.571       0.558      0.566      0.516
## 196     100.000     100.000       1.000      0.980      0.981
## 197     100.000     100.000       1.000      0.928      0.962
## 198     100.000     100.000       1.000      0.918      0.870
## 199      95.082      90.714       0.960      0.958      0.971
## 200      95.082      90.714       0.858      0.823      0.962
## 201      90.164      95.000       0.848      0.755      0.870
## 202     100.000      67.857       0.752      0.740      0.514
## 203     100.000      67.143       0.671      0.683      0.462
## 204      98.361      68.571       0.558      0.566      0.516
## 205     100.000     100.000       1.000      0.978      0.981
## 206     100.000     100.000       1.000      0.895      0.962
## 207     100.000     100.000       1.000      0.867      0.870
## 208      97.701      89.000       0.956         NA      0.971
## 209      97.701      89.000       0.867         NA      0.962
## 210      97.701      89.000       0.814         NA      0.870
## 211      98.851      69.000       0.749         NA      0.510
## 212      98.851      68.500       0.679         NA      0.462
## 213      98.851      68.500       0.566         NA      0.516
## 214     100.000     100.000       1.000         NA      0.981
## 215     100.000     100.000       1.000         NA      0.962
## 216     100.000     100.000       1.000         NA      0.870
## 217      97.701      85.786       0.956         NA      0.971
## 218      97.701      85.535       0.832         NA      0.962
## 219      88.506      92.830       0.656         NA      0.870
## 220      98.851      73.082       0.806         NA      0.505
## 221      98.851      73.082       0.719         NA      0.462
## 222      98.851      73.082       0.344         NA      0.516
## 223     100.000     100.000       1.000         NA      0.952
## 224     100.000     100.000       1.000         NA      0.923
## 225     100.000     100.000       1.000         NA      0.762
# Select TSS
modelEvals <- modelEvals |>
  dplyr::filter(metric.eval == "TSS") |>
  dplyr::group_by(algo) |>
  dplyr::reframe(TSS = evaluation,
                 name = full.name)

# Plot TSS
ggplot(data = modelEvals, mapping = aes(x = algo, y = TSS)) +
  geom_boxplot() +
  labs(x = "Model") +
  theme_bw()

Variable contributions

The contributions of different variables are also in the output model object. Last time I checked the contributions were not relative, but that may have changed.

# Extract variable importance
varImportance <- get_variables_importance(obj = modelOut)

# Plot variable importance
ggplot(data = varImportance, mapping = aes(x = algo, y = var.imp)) +
  geom_boxplot() +
  facet_wrap(~expl.var) +
  labs(x = "Model") +
  theme_bw()

Response curves

Response curves show the predicted values of the response across the range of each environmental variable. Checking the response curves is a nice way to see what relationships the model may be picking up on, and to look at the ecological plausiblity of a model. If you get a really wacky response curve, it’s a good idea to go back and check that model. It might mean that the model does not meet the right statistical assumptions or a term is insignificant.

# Select highest performing GLMs
select_models <- modelEvals[order(-modelEvals$TSS),] |>
  dplyr::filter(algo == "GLM") |>
  head(5)

# GLM response curves
par(mar=c(3,3,3,3))
bm_PlotResponseCurves(bm.out = modelOut,
                      models.chosen = select_models$name,
                      new.env = get_formal_data(modelOut, "expl.var"),
                      show.variables = get_formal_data(modelOut, "expl.var.names"),
                      fixed.var = "mean",
                      do.bivariate = FALSE,
                      do.plot = TRUE,
                      do.progress = TRUE)
## No id variables; using all as measure variables

# Select highest performing GAMs
select_models <- modelEvals[order(-modelEvals$TSS),] |>
  dplyr::filter(algo == "GAM") |>
  head(5)

# GAM response curves
par(mar=c(3,3,3,3))
bm_PlotResponseCurves(bm.out = modelOut,
                      models.chosen = select_models$name,
                      new.env = get_formal_data(modelOut, "expl.var"),
                      show.variables = get_formal_data(modelOut, "expl.var.names"),
                      fixed.var = "mean",
                      do.bivariate = FALSE,
                      do.plot = TRUE,
                      do.progress = TRUE)
## No id variables; using all as measure variables

# Select highest performing RFs
select_models <- modelEvals[order(-modelEvals$TSS),] |>
  dplyr::filter(algo == "GLM") |>
  head(5)

# RF response curves
par(mar=c(3,3,3,3))
bm_PlotResponseCurves(bm.out = modelOut,
                      models.chosen = select_models$name,
                      new.env = get_formal_data(modelOut, "expl.var"),
                      show.variables = get_formal_data(modelOut, "expl.var.names"),
                      fixed.var = "mean",
                      do.bivariate = FALSE,
                      do.plot = TRUE,
                      do.progress = TRUE)
## No id variables; using all as measure variables

Project models onto summer 2015

Finally, we will project our model onto the environmental data from 2015 (the year we left out). To “quality-control” the projections we will only use the top 5 performing models for each algorithm. This is also a good chance to see how predictions from different model formulas differ.

# Project models onto year 2015 (witheld from model)
env_proj <- raster::subset(env_dat, c(grep('2015', names(env_dat))))
names(env_proj) <- vars

# Select highest performing GLMs
select_models <- modelEvals[order(-modelEvals$TSS),] |>
  dplyr::filter(algo == "GLM") |>
  head(5)

glmProj <- BIOMOD_Projection(bm.mod = modelOut,
                            new.env = env_proj,
                            proj.name = "GLM",
                            models.chosen = select_models$name,
                            metric.binary = "TSS",
                            compress = TRUE)
## 
## -=-=-=-=-=-=-=-=-=-=-=-=-= Do Single Models Projection -=-=-=-=-=-=-=-=-=-=-=-=-=
## 
##  > Building clamping mask
## 
##  > Projecting RightWhale_PA1_RUN1_GLM ...
##  > Projecting RightWhale_PA1_RUN2_GLM ...
##  > Projecting RightWhale_PA1_RUN3_GLM ...
##  > Projecting RightWhale_PA1_RUN4_GLM ...
##  > Projecting RightWhale_PA1_RUN5_GLM ...
## 
##  > Building TSS binaries / filtered
## 
## -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Done -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
# Select highest performing GAMs
select_models <- modelEvals[order(-modelEvals$TSS),] |>
  dplyr::filter(algo == "GAM") |>
  head(5)

gamProj <- BIOMOD_Projection(bm.mod = modelOut,
                             new.env = env_proj,
                             proj.name = "gam",
                             models.chosen = select_models$name,
                             metric.binary = "TSS",
                             compress = TRUE)
## 
## -=-=-=-=-=-=-=-=-=-=-=-=-= Do Single Models Projection -=-=-=-=-=-=-=-=-=-=-=-=-=
## 
##  > Building clamping mask
## 
##  > Projecting RightWhale_PA1_RUN1_GAM ...
##  > Projecting RightWhale_PA1_RUN2_GAM ...
##  > Projecting RightWhale_PA1_RUN3_GAM ...
##  > Projecting RightWhale_PA1_RUN4_GAM ...
##  > Projecting RightWhale_PA1_RUN5_GAM ...
## 
##  > Building TSS binaries / filtered
## 
## -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Done -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
# Select highest performing RFs
select_models <- modelEvals[order(-modelEvals$TSS),] |>
  dplyr::filter(algo == "RF") |>
  head(5)

rfProj <- BIOMOD_Projection(bm.mod = modelOut,
                             new.env = env_proj,
                             proj.name = "RF",
                             models.chosen = select_models$name,
                             metric.binary = "TSS",
                             compress = TRUE)
## 
## -=-=-=-=-=-=-=-=-=-=-=-=-= Do Single Models Projection -=-=-=-=-=-=-=-=-=-=-=-=-=
## 
##  > Building clamping mask
## 
##  > Projecting RightWhale_PA1_RUN1_RF ...
##  > Projecting RightWhale_PA1_RUN2_RF ...
##  > Projecting RightWhale_PA1_RUN3_RF ...
##  > Projecting RightWhale_PA1_RUN4_RF ...
##  > Projecting RightWhale_PA1_allRun_RF ...
## 
##  > Building TSS binaries / filtered
## 
## -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Done -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
plot(glmProj)
## Loading required namespace: tidyterra

plot(gamProj)

plot(rfProj)

Looking a little further into the projection objects (using GAM as an example):

# Look closer at GAM projection
gamProj
## 
## -=-=-=-=-=-=-=-=-=-=-=-=-=-= BIOMOD.projection.out -=-=-=-=-=-=-=-=-=-=-=-=-=-=
## 
## Projection directory : ./RightWhale/gam
## 
## 
## sp.name : RightWhale
## 
## expl.var.names : Depth SST CHL
## 
## 
## modeling.id : RightWhale ( ./RightWhale/RightWhale.RightWhale.models.out )
## 
## models.projected : 
## RightWhale_PA1_RUN1_GAM, RightWhale_PA1_RUN2_GAM, RightWhale_PA1_RUN3_GAM, RightWhale_PA1_RUN4_GAM, RightWhale_PA1_RUN5_GAM
## 
## available binary projection : TSS
## 
## -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
fp <- gamProj@proj.out@link[1]

# Load in as SpatRaster
gamStack <- terra::rast(fp)

# fetch right whale sightings
whales2015 <- RightWhale |>
  dplyr::filter(year == 2015)

# Plot highest performing GAM (slot 1)
plot(gamStack$RightWhale_PA1_RUN1_GAM) 
points(x = whales2015$lon, y = whales2015$lat, cex = 2, pch = 16) # Add whale sightings